The spatial independence and temporal continuity of video data as a whole are not fully investigated for video action detection. To tackle this issue, a deep network architecture is proposed, named Pseudo-3D Convolutional Tube Network (P3D-CTN). In particular, the proposed P3D-CTN integrates the frame-based two-dimensional convolutional module with the P3D convolutional module to balance the spatial and temporal information, and generates deeper features about human actions. Evaluations on two benchmark datasets (i.e., UCF-Sports and J-HMDB) demonstrate that the proposed P3D-CTN has superior performances in the task of action label prediction and yields state-of-the-art results for spatio-temporal action detection.