Recent literature has proposed new methods for measuring the systemic risk of financial institutions based on observed stock returns. In this paper we examine the reliability and robustness of such risk measures, focusing on CoVaR, marginal expected shortfall, and option-based tail risk estimates. We show that CoVaR exhibits undesired characteristics in the way it responds to idiosyncratic risk. In the presence of contagion, the risk measures provide conflicting signals on the systemic risk of infectious and infected banks. Finally, we explore how limited data availability typical of practical applications may limit the measures’ performance. We generate systemic tail risk through positions in standard index options and describe situations in which systemic risk is misestimated by the three measures. The observations raise doubts about the informativeness of the proposed measures. In particular, a direct application to regulatory capital surcharges for systemic risk could create wrong incentives for banks.