Skills-based learning environments are used to promote the acquisition of practical skills as well as decision making, communication, and problem solving. It is important to provide feedback to the students from these sessions and observations of their actions may inform the assessment process and help researchers to better understand the learning process. Through a series of prototype demonstrators, we have investigated the use of semantic annotation in the recording and subsequent understanding of such simulation environments. Our Semantic Web approach is outlined and conclusions drawn as to the suitability of different annotation methods and their combination with ubiquitous computing techniques to provide novel mechanisms for both student feedback and increased understanding of the learning environment.