IEC 62351 is an industry standard aimed at improving security in automation systems in the power system domain. It contains provisions to ensure the integrity, authenticity and confidentiality for different protocols used in power systems. In this paper we look at the different parts of IEC 62351 and assess to what extent the standard manages to improve security in automation systems. We also point out some incongruities in the algorithms or parameters chosen in parts of the standard. Overall, we conclude that the standard can significantly improve security in power systems if applied comprehensively, but we also note that the need to preserve (partial) backwards-compatibility has led to some design choices that provide less security than could have been achieved with a more ambitious approach.