Human-in-the-loop robot learning allows a robot to learn tasks more effectively with the help of humans in the role of teacher. While there is a large body of work on algorithms that leverage human input for better robot learning, there has been little attention to understanding how humans teach robots. In this paper, we provide preliminary results on how users strategize the use of demonstrations and evaluative feedback under a budget, and how these choices are influenced by demographic variables such as gender. We implemented a learning algorithm that allows a simulated robot arm to learn three reaching tasks with the help of a human. We collected interaction data for a total of 58 participants, which shows that participants demonstrate a tendency to provide evaluative feedback earlier in their interactions compared to demonstrations, and that gender may have an influence on teaching strategy. This preliminary analysis lays the foundation for future research aimed at developing tuneable computational models of different human teachers.