Boxing punch type classification and kinematic analysis are essential for coaches and athletes, providing critical insights into punch variety and effectiveness, which are vital for performance improvement. Existing methods for punch recognition and classification typically rely on wearable sensor data or video data; however, no fully automated system currently exists. While coaches prefer video-based analysis for its ability to easily visualize punch action errors and refine technique, video-based classification suffers from lower accuracy compared to sensor-based methods due to limitations such as motion blur. Current classification approaches typically employ supervised learning, requiring experts to annotate 70–80% of the data for model training. However, the high sampling frequency of sensor data makes this process time-consuming and challenging, leading to potential fatigue and an increased risk of inconsistent annotations by domain experts. This paper proposes a novel multimodal approach that integrates wearable sensor data and video data for automatic punch recognition and classification. The method also includes automatic segmentation of punch videos, which improves classification accuracy by utilizing both data sources. To reduce labeling effort, we apply a Query by Committee-based active learning technique, significantly decreasing the required labeling effort by one-sixth. Using only 15% of the typical labeling effort, our system achieves 91.41% accuracy for rear-hand punch recognition, 91.91% for lead-hand punch recognition, and 92.33% and 94.56% for punch classification, respectively. This Smart Boxer system aims to enhance punch analytics in boxing, providing valuable insights to improve training, optimize performance, and increase fan engagement with the sport.