Abstract Structural damage detection using vibration-based techniques and transmissibility is investigated and proposed with elaborating the mathematical interrelationship between modal assurance criterion (MAC) and cosine similarity measure. From this interrelationship, two new damage indicators, namely cosine based indicator (CI) and extended transmissibility damage indicator (ETDI), are drawn out. Even MAC and cosine similarity measure are normally adopted separately; their kernel agrees well with each other. In this study, we extend these two indicators, derive two new damage indicators, CI and ETDC, and apply them in a transmissibility based structural damage detection technique. In order to verify the feasibility of the developed damage indicators, experimental data for a three-story benchmark structure is adopted. The results from the benchmark demonstrate well performance of both indicators in damage detection, which might imply their potential applicability in further real engineering structures.