With the state of the art for many MIR tasks being based in deep learning, it is crucial that there be a large amount of data to train these models. Many approaches opt for audio that has been synthesized from MIDI data, but these often lack the musicality and nuances of recorded musicians. We present QuartSet, a dataset of real-instrument recordings and their accompanying scores for the tasks of audio-to-score transcription and score-informed source separation. The scores of QuartSet are provided in Kern notation, which is a simple format that is easily manipulable. Instead of being simplified, as is the common practice for transcription works, the Kern scores contain every dynamic, articulation, and ornament marking as written by the original composers. This is so the scores fully reflect what is present in the audio recordings. We show that QuartSet can be easily integrated with MIDI-synthesized data to create a larger, more diverse dataset to train a transcription model. When trained on both synthesized and real audio data, the model was able to produce better transcriptions of other real audio than the model that was trained only on synthesized data. Finally, we suggest potential methods for creating more audio and score data without synthesis.