CAR-T cell therapy is at the forefront of next-generation multiple myeloma (MM) management, with two B-cell maturation antigen (BCMA)-targeted products recently approved. However, these products are incapable of breaking the infamous pattern of patient relapse. Two contributing factors are the use of BCMA as a target molecule and the artificial scFv format that is responsible for antigen recognition. Tackling both points of improvement in the present study, we used previously characterized VHHs that specifically target the idiotype of murine 5T33 MM cells. This idiotype represents one of the most promising yet challenging MM target antigens, as it is highly cancer- but also patient-specific. These VHHs were incorporated into VHH-based CAR modules, the format of which has advantages compared to scFv-based CARs. This allowed a side-by-side comparison of the influence of the targeting domain on T cell activation. Surprisingly, VHHs previously selected as lead compounds for targeted MM radiotherapy are not the best (CAR-) T cell activators. Moreover, the majority of the evaluated VHHs are incapable of inducing any T cell activation. As such, we highlight the importance of specific VHH selection, depending on its intended use, and thereby raise an important shortcoming of current common CAR development approaches.