Many protein kinases are activated through phosphorylation of an activation loop thereby turning on downstream signaling pathways. Activation of JAK2, a nonreceptor tyrosine kinase with an important role in growth factor and cytokine signaling, requires phosphorylation of the 1007 and 1008 tyrosyl residues. Dephosphorylation of these two sites by phosphatases presumably inactivates the enzyme, but the underlying mechanism is not known. In this study, we employed MALDI‐TOF/TOF and triple quadrupole mass spectrometers to analyze qualitatively and quantitatively the dephosphorylation process by using synthetic peptides derived from the tandem autophosphorylation sites (Y1007 and Y1008) of human JAK2. We found that tyrosine phosphatases catalyzed the dephosphorylation reaction sequentially, but different enzymes exhibited different selectivity. Protein tyrosine phosphatase 1B caused rapid dephosphorylation of Y1008 followed by Y1007, while SHP1 and SHP2 selectively dephosphorylated Y1008 only, and yet HePTP randomly removed a single phosphate from either Y1007 or Y1008, leaving behind mono‐phosphorylated peptides. The specificity of dephosphorylation was further confirmed by molecular modeling. The data reveal multiple modes of JAK2 regulation by tyrosine phosphatases, reflecting a complex, and intricate interplay between protein phosphorylation and dephosphorylation.