This paper presents a new system for teachers' natural complex action recognition in the smart classroom in order to realize an intelligent cameraman and virtual mouse. First, the system proposes a hybrid human model and employs a 2-order B-spline function to detect the two shoulder joints in the silhouette image to obtain the basic motion features including the elbow angles, motion parameters of the face and two hands. Then, a primitive-based coupled hidden Markov model (PCHMM) is presented for natural context-dependent action recognition. Last, some comparison experiments show that PCHMM is better than the traditional HMM and coupled HMM.