To analyze the complexity of continuous chaotic systems better, the modified multiscale permutation entropy (MMPE) algorithm is proposed. Characteristics and parameter choices of the MMPE algorithm are investigated. The comparative study between MPE and MMPE shows that MMPE has better robustness for identifying different chaotic systems when the scale factor τ takes large values. Compared with MPE, MMPE algorithm is more suitable for analyzing the complexity of time series as it has τ time series. For its application, MMPE algorithm is used to calculate the complexity of multiscroll chaotic systems. Results show that complexity of multiscroll chaotic systems does not increase as scroll number increases. Discussions based on first‐order difference operation present a reasonable explanation on why the complexity does not increase. This complexity analysis method lays a theoretical as well as experimental basis for the applications of multiscroll chaotic systems. © 2014 Wiley Periodicals, Inc. Complexity 21: 52–58, 2016