Quality assessment of jazz improvisation is a multi-faceted, high-level cognitive task routinely performed by educators in university jazz programs and other discriminating music listeners. In this pilot study, we present a novel dataset of 88 MIDI jazz piano improvisations with ratings of creativity, technical proficiency, and aesthetic appeal provided by four jazz experts, and we detail the design of a feature set that can represent some of the rhythmic, melodic, harmonic, and other expressive attributes humans recognize as salient in assessment of performance quality. Inherent subjectivity in these assessments is inevitable, yet the recognition of performance attributes by which humans perceive quality has wide applicability to related tasks in the music information retrieval (MIR) community and jazz pedagogy. Preliminary results indicate that several musiciologically-informed features of relatively low computational complexity perform reasonably well in predicting performance quality labels via ordinary least squares regression.