Abstract Autogram is an effective optimal frequency band selection method, in which the signal spectrum is divided by the maximum overlapping discrete wavelet packet transform (MODWPT) and the position of maximum kurtosis value is used as the optimal frequency band. However, Autogram follows a binary tree structure in segmenting frequency domain and its segmentation position is fixed, this causes that its position cannot be adaptively determined according to the signal characteristics. To solve this issue, in this paper an improved frequency band selection method called maximum envelope based-Autogram (MEAutogram) is proposed. In MEAutogram method, the maximum value envelope method is used to process the signal spectrum and then the minimum value point closest to the middle position of adjacent maximum value points in envelope signal is used as the segmentation position. However, the segmentation accuracy of MEAutogram will decrease when the signal contains lots of irrelevant components. The recently proposed nonlinear time series analysis method termed symplectic geometry mode decomposition (SGMD) founded on the symplectic matrix similar transformation is used to remove irrelevant components. Based on this, a new SGMD and MEAutogram based fault diagnosis method for gear is proposed. The proposed fault diagnosis method of gear can reduce the calculation amount through overcoming the influence of irrelevant components on the segmentation position, which can be adaptively determined according to the characteristics of the raw signal. Finally, the analysis results of simulation and gear test data was used to verify that the appropriate demodulation frequency band can be accurately detected by the proposed method and the fault characteristics obtained by the proposed method are more obvious than that of the comparative methods.