The global dissemination of plasmid-mediated colistin resistance gene mcr and its variants have posed a great threat to public health. Therefore, the Chinese government banned the use of colistin as a feed additive in livestock in April 2017. To explore the dynamic changes of overall antibiotic resistance genes (ARGs) and phylogenetic relationship of bacteria from a single pig farm before and after the withdrawal of colistin, fecal swab samples were collected from a large-scale pig farm before (n = 32; 2 months pre-withdrawal of colistin) and after withdrawal of colistin (n = 30; 13 months post-withdrawal of colistin). Escherichia coli and Klebsiella pneumoniae were isolated. Whole-genome sequencing (Illumina, MiSeq) was performed to examine ARGs, plasmids and the genetic relationship of the isolates. The overall SNP results indicated all isolates had high genetic diversity, and the evolutionary relationship across isolates was not influenced by the ban of colistin. However, the prevalence of mcr-1.1 (5.6%, p < 0.01) was significantly lower than before the ban (86.4%). Plasmid profiling analysis showed that 17 of 20 (85.0%) observed mcr-1.1 genes reside on IncX4-type plasmids, 16 of which (94.1%) were from isolates before the ban. On the contrary, the presence of blaCTX-M gene was significantly increased (p = 0.0215) post-withdrawal of colistin. Our results showed that withdrawal of colistin reduced the incidence of mcr-1-harboring IncX4-type plasmids, but had limited influences on unrelated ARGs.