For bridging the semantic gap between the low-level features of videos and high-level semantic concepts in videos, we propose a multi-semantic video annotation method with semantic network. First, we use the semantic network to represent the high-level semantic knowledge and model the relationships between the concepts. Then we divide the videos to key frames and use Convolutional Neural Networks (CNNs) to extract low-level visual features and detect the concepts in the videos. Finally, we combine the low-level features with the high-level knowledge to perform a two-level reasoning to optimize the result. Experiment results show that the proposed method significantly outperforms existing video annotation techniques in terms of precision value.