Neuropsychiatric disorder (ND) is often accompanied by abnormal functional connectivity (FC) patterns in specific task contexts. The distinctive task-specific FC patterns can provide valuable features for ND classification models using deep learning. However, most previous studies rely solely on the whole-brain FC matrix without considering the prior knowledge of task-specific FC patterns. Insight by the decoding studies on brain-behavior relationship, we develop TSP-GNN, which extracts task-specific prior (TSP) connectome patterns and employs graph neural network (GNN) for disease classification. TSP-GNN was validated using publicly available datasets. Our results demonstrate that different ND types show distinct task-specific connectivity patterns. Compared with the whole-brain node characteristics, utilizing task-specific nodes enhances the accuracy of ND classification. TSP-GNN comprises the first attempt to incorporate prior task-specific connectome patterns and the power of deep learning. This study elucidates the association between brain dysfunction and specific cognitive processes, offering valuable insights into the cognitive mechanism of neuropsychiatric disease.