Generative models in deep learning have experienced great development in art generation. Even though image-based art generation has had a big success, music still needs to catch up compared to its visual counterpart. Extreme focus on improving the generated outputs has neglected the importance of understanding what generative models learn and understand about music. This investigation aims to understand how latent space characteristics can be related to the concepts of creativity, fidelity, and novelty. Unsupervised variational autoencoders (VAEs) with different latent space characteristics were trained to generate chords. Reconstruction and generation capabilities were analyzed. A set of probing networks was trained to determine the representations learned by the unsupervised models. Particular focus was drawn to identify which musical concepts were learned in the latent space. Analysis shows that a bigger latent space will favor, with limitations, novelty-creativity at the expense of fidelity-standards, which gets worse but also to a limit. Other findings show that smaller latent spaces do not allow for good dataset reconstruction but still follow good fidelity-standards at generation time at the expense of lower novelty-creativity. Finally, results show that bigger latent spaces are required for learning complex musical concepts.