Side-channel attacks and fault attacks use the physical interactions with cryptographic devices to achieve the shortcuts of recovering the secret key. For side-channel attacks, physical information leakage during the cryptographic calculation is measured and statistically analyzed to reveal the intermediate values. For fault attacks, computational faults are intentionally triggered to obtain faulty outputs or fault behaviors for the key recovery. Fault injections are also used to obtain special information leakage under the faulty environment. This paper gives a general overview of these physical attacks and discusses the necessity of the accurate complexity evaluation of their successful key recovery.