Exploitation of multiplayer interaction and development of virtual puppetry storytelling using gesture control and stereoscopic devices

Article English OPEN
Liang, Hui ; Chang, Jian ; Deng, Shujie ; Chen, C. ; Tong, R. ; Zhang, Jian J. (2017)

With the rapid development of human-computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head-mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head-mounted display outperformed the traditional monoscopic display in a comparison between the two.
  • References (42)
    42 references, page 1 of 5

    1. Psomos P, Kordaki M. Pedagogical analysis of educational digital storytelling environments of the last five years. ProcediaSocial and Behavioural Sciences 2012; 46: 1213-1218.

    2. More C. Digital stories targeting social skills for children with disabilities multidimensional learning. Intervention in School and Clinic 2008; 43(3): 168-177.

    3. Garzotto F, Paolini P, Sabiescu A. Interactive storytelling for children. In Proceedings of the 9th International Conference on Interaction Design and Children, ACM 2010; 356-359.

    4. Hofer M, Owings SWAN K. Digital storytelling: Moving from promise to practice. In Society for Information Technology & Teacher Education International Conference, 2006(1): 679-684.

    5. Lu F, Tian F, Jiang Y, Cao X, Luo W, Li G, Zhang X, Dai G, Wang H. ShadowStory: creative and collaborative digital storytelling inspired by cultural heritage. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM 2011; 1919-1928.

    6. Thompson P. The digital natives as learners: Technology use patterns and approaches to learning. Computers & Education 2013; 65: 12-33.

    7. Barnes K, Marateo RC, Ferris SP. Teaching and learning with the net generation. Innovate: Journal of Online Education 2007; 3(4): 1.

    8. Jonassen DH. Computers as mind tools for schools: Engaging critical thinking, Prentice Hall, 2000.

    9. Smeets E. Does ICT contribute to powerful learning environments in primary education? Computers & Education 2005; 44(3): 343-355.

    10. Garzotto F, Forfori M. FaTe2: storytelling edutainment experiences in 2D and 3D collaborative spaces. In Proceedings of the 2006 conference on Interaction design and children, ACM 2006; 113-116.

  • Metrics
    No metrics available
Share - Bookmark