This paper presents two strategies to prevent the pitch embeddings from being too close to the dataset characteristics so as to improve the pitch and pitch class distributions of generation. The first strategy is to switch the pitch representation from the MIDI number representation to an alternative representation that encodes a pitch into pitch class and octave, which forces musically similar pitches to share part of the embedding vectors. The second strategy freezes the pitch embeddings during training according to the proposed metrics that evaluate the quality of pitch embedding space, maintaining the advantage of the embedding obtained in the first strategy. The experiments show that, when both strategies are applied on the training in an auto-regressive melody generation task, the generated samples exhibit slightly improved pitch and noticeably improved pitch class distributions, indicating the effectiveness of both strategies.