This research aims to develop a wearable musical interfacewhich enables to control audio and video signals by usinghand gestures and human body motions. We have beendeveloping an audio-visual manipulation system that realizes tracks control, time-based operations and searching fortracks from massive music library. It aims to build an emotional and affecting musical interaction, and will providea better method of music listening to people. A sophisticated glove-like device with an acceleration sensor and several strain sensors has been developed. A realtime signalprocessing and musical control are executed as a result ofgesture recognition. We also developed a stand-alone device that performs as a musical controller and player at thesame time. In this paper, we describe the development of acompact and sophisticated sensor device, and demonstrateits performance of audio and video signals control.