Sequence matching/alignment based scheme has been common for action recognition. Such a typical scheme, however, requires tremendous amounts of computation when the volume of prototypical action videos is large and easily causes mismatching for border-isolated samples in action categories. In this paper, we propose a framework of averaging video sequences based on multi-dimensional dynamic time warping (MD-DTW) and propose to use the resulting average actions, instead of prototypical action videos, for action recognition. Experimental results show that 1) average actions were shown to be more discriminative than prototypical video sequences for action modeling, and 2) action recognition using average actions rather than using prototypical action videos is much more efficient and advanced.