Off-target toxicity is a major cause of dose-limiting toxicity for antibody-drug conjugates (ADCs), mechanisms of which remain poorly understood. Here, we demonstrate that cytoskeleton-associated protein 5 (CKAP5) serves as a cell surface target for T-DM1 and that binding of T-DM1 to CKAP5 is mediated by payload (DM1). This study introduces a novel molecular mechanism of ADC payload-mediated interaction with cell surface molecules to induce cytotoxicity. Upon binding to CKAP5, T-DM1 causes cell membrane damage and leads to calcium influx into the cells, resulting in disorganized microtubule network and apoptosis. While binding of T-DM1 with HER2 is critical for killing HER2-positive tumor cells, our data suggest that cytotoxicity induced by T-DM1 interaction with CKAP5 may preferentially damage normal cells/tissues where HER2 expression is low or missing to cause off-target toxicity. This study provides molecular basis of ADC-induced off-target cytotoxicity and opens a new avenue for developing next generation of ADCs.