A method of audio matrix decoding in which a moving sound image is restored includes decoding multichannel signals from stereo signals, extracting strengths and positions of virtual sound sources existing between channels based on power vectors of the decoded multichannel signals, comparing the strengths and positions of the extracted previous and current virtual sound sources to predict position movement and the strengths of the virtual sound sources, and redistributing powers to positions of channel speakers in a multichannel arrangement based on the predicted position of a sound image.