AbstractA novel adaptive weighted geometric‐shaped (GS) 61 quadrature amplitude modulation (QAM) constellation is proposed in order to mitigate the nonlinear damage and increase the transmission distance. By introducing an adaptive weighting factor related to Kerr effect and fiber transmission distance in concentric regular hexagon constellation, the overlapping problem of the outer constellation points induced by nonlinear damage was reduced. The simulation results show that the proposed GS‐61QAM outperformed uniform 64QAM and has more obvious advantages in long‐haul optical transmission system. For a 120‐Gb/s coherent optical communication system, compared with uniform 64QAM, the proposed algorithm can obtain about 4.1 dB optical signal‐to‐noise (OSNR) gain at the BER of 10−2. Moreover, in order to further improve the system tolerance of nonlinearity, a scheme of adaptive weighted GS‐61QAM in coherent optical communication system utilizing K‐Means algorithm is proposed, the proposed scheme outperforms the traditional uniform 64QAM communication system by 5 dB in OSNR at the BER of 10−2.