We identify a few sample eigenvalues adjustment patterns that lead to an improvement in the out-of-sample portfolio Sharpe ratio when the population covariance matrix admits a high-dimensional factor model. These patterns unveil the key to portfolio performance improvement and shed light on the effectiveness of a few well-known covariance matrix estimation methods which were not designed to improve the out-of-sample portfolio performance.