Cancer cell plasticity is the ability of neoplastic cells to alter their identity and acquire new biological properties under microenvironmental pressures. In prostate cancer (PCa), lineage plasticity often results in therapy resistance and trans-differentiation to neuroendocrine (NE) lineage. However, identifying the cancer cells harboring lineage plasticity-related status remains challenging. Here, we established a lineage plasticity-related gene signature (LPSig) using 3314 PCa samples from 13 multi-center public transcriptomic datasets of human PCa. Based on LPSig, we identified a previously undefined minority population of lineage plasticity-related PCa cells (LPCs) from a comprehensive single-cell RNA-sequencing (scRNA-seq) meta-atlas consisted of 8 public human PCa scRNA-seq cohorts. Furthermore, in-depth dissection revealed pivotal roles of LPCs in trans-differentiation, tumor recurrence, and poor patient survival during PCa progression. Furthermore, we identified HMMR as a representative cell surface marker for LPCs, which was validated using additional scRNA-seq datasets and multiplexed immunohistochemistry. Moreover, HMMR was transcriptionally inhibited by AR, and was required for the aggressive adenocarcinoma features and NE phenotype. Collectively, our study uncovers a novel population of lineage plasticity-related cells with low AR activity, stemness-like traits, and elevated HMMR expression, that may facilitate poor prognosis in PCa.