Recent developments in generative AI have posed a challenge for developers who attempt to maintain an effective balance between the system's generative input and user's sense of creativity and control. In this paper we present a longitudinal study of a web/mobile application we developed titled Mixboard, which allows novices and music lovers to create and share personalized musical mashups in a co-creative manner. Different balances between the role of system automation and user creative input have been developed and studied over a period of two years. Findings from a set of the longitudinal users studies indicate that while novices appreciate the system's AI driven automation and suggestion, the more they get familiar with the system they seek to expand their level of control and creative input into the final product. Future developments may therefore include a personalized level of control balance based on continuous assessment of user behaviour.